Raw memory transaction support

ABSTRACT

Methods, systems, and apparatus for implementing raw memory transactions. An SoC is configured with a plurality of nodes coupled together forming a ring interconnect. Processing cores and memory cache components are operatively coupled to and co-located at respective nodes. The memory cache components include a plurality of last level caches (LLC&#39;s) operating as a distributed LLC and a plurality of home agents and caching agents employed for supporting coherent memory transactions. Route-back tables are used to encode memory transactions requests with embedded routing data that is implemented by agents that facilitate data transfers between link interface nodes and memory controllers. Accordingly, memory request data corresponding to raw memory transactions may be routed back to requesting entities using headerless packets.

FIELD OF THE INVENTION

The field of invention relates generally to computer system interfacesand, more specifically but not exclusively relates to techniques forfacilitating raw memory transactions for System on a Chip (SoC)architectures.

BACKGROUND INFORMATION

Computer systems typically employ one or more interconnects tofacilitate communication between system components, such as betweenprocessors and memory. Interconnects and/or expansion interfaces mayalso be used to support built-in and add on devices, such as IO(input/output) devices and expansion cards and the like. For many yearsafter the personal computer was introduced, the primary form ofinterconnect was a parallel bus. Parallel bus structures were used forboth internal data transfers and expansion buses, such as ISA (IndustryStandard Architecture), MCA (Micro Channel Architecture), EISA (Extendedindustry Standard Architecture) and VESA Local Bus. In the early 1990'sIntel Corporation introduced the PCI (Peripheral Component Interconnect)computer bus. PCI improved on earlier bus technologies by not onlyincreasing the bus speed, but also introducing automatic configurationand transaction-based data transfers using shared address and datalines.

As time progressed, computer processor clock rates where increasing at afaster pace than parallel bus clock rates. As a result, computerworkloads were often limited by interconnect bottlenecks rather thanprocessor speed. Although parallel buses support the transfer of a largeamount of data (e.g., 32 or even 64 bits under PCI-X) with each cycle,their clock rates are limited by timing skew considerations, leading toa practical limit to maximum bus speed. To overcome this problem,high-speed serial interconnects were developed. Examples of early serialinterconnects include Serial ATA, USB (Universal Serial Bus), FireWire,and RapidIO.

Another standard serial interconnect that is widely used is PCI Express,also called PCIe, which was introduced in 2004 under the PCIe 1.0standard. PCIe was designed to replace older PCI and PCI-X standards,while providing legacy support. PCIe employs point-to-point serial linksrather than a shared parallel bus architecture. Each link supports apoint-to-point, communication channel between two PCIe ports using oneor more lanes, with each lane comprising a bi-directional serial link.The lanes are physically routed using a crossbar switch architecture,which supports communication between multiple devices at the same time.As a result of its inherent advantages, PCIe has replaced PCI as themost prevalent interconnect in today's personal computers. PCIe is anindustry standard managed by the PCI-SIG (Special Interest Group). Assuch, PCIe pads are available from many ASIC and silicon vendors.

Recently, Intel introduced the QuickPath Interconnect® (QPI). QPI wasinitially implemented as a point-to-point processor interconnectreplacing the Front Side Bus on platforms using high-performanceprocessors, such as Intel® Xeon®, and Itanium® processors. QPI isscalable, and is particularly advantageous in systems having multipleprocessors employing shared memory resources. QPI transactions employpacket-based transfers using a multi-layer protocol architecture. Amongits features is support for coherent transaction (e.g., memorycoherency).

A significant amount of I/O bandwidth is consumed by memorytransactions. One approach that is currently employed to increase memorytransaction bandwidth is to employ a Fully Buffered DIMM (or FB-DIMM)architecture, which introduces an advanced memory buffer (AMB) between amemory controller and a memory module. Unlike the parallel busarchitecture of traditional DRAMs, an FB-DIMM has a serial interfacebetween the memory controller and the AMB. This enables an increase tothe width of the memory without increasing the pin count of the memorycontroller beyond a feasible level. With this architecture, the memorycontroller does not write to the memory module directly; rather it isdone via the AMB. The AMB can thus compensate for signal deteriorationby buffering and resending the signal. In addition, the AMB can alsooffer error correction, without posing an overhead on the processor orthe memory controller. In addition to BF-DIMM. Intel has recentlyintroduced the Intel® Scalable Memory Interconnect (SMI) and ScalableMemory Buffers (SMB). The integrated SMI offers high-speed serial linksto the SMBs, which support cost-effective, commodity RDDR3 memory.

Other recent advancements include multi-core processors and System on aChip (SoC) architectures. Rather than interfacing discrete components ona printed circuit board or through use of other package configurations,on an SoC multiple components are integrated onto a single integratedchip. SoCs offer a number of advantages, including denser packaging,higher speed communication between functional components, and lowertemperature operation. SoC designs also provide standardization,scalability, modularization, and reusability.

Although SoC architectures are clearly the future direction of systemdesigns, there are instances where it is still advantageous to keep somecomponents on separate chips or dies. For example, it may beadvantageous to have components with dedicated or specialized functions,such as memory controllers, on separate chips. At the same time, itwould be desirable to support data transfer rates with such externalcomponents as if they were integrated on the SoC.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same becomesbetter understood by reference to the following detailed description,when taken in conjunction with the accompanying drawings, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified:

FIG. 1 is a block schematic diagram of a system architecture depictingselected components of an SoC and various external interfaces;

FIG. 2 a is a block diagram illustrating selected components from thesystem architecture of FIG. 1 related to supporting memory transactionsvia use of distributed caching agents;

FIG. 2 b is a block diagram depicting an augmentation to thearchitecture of FIG. 2 a further including the use of distributed homeagents that are operatively coupled to memory controllers via KTI links;

FIG. 3 a is a Hock schematic diagram of an SoC architecture employing aring interconnect and implementing the distributed caching agentarchitecture of FIG. 2 a;

FIG. 3 b is a block schematic diagram of an SoC architecture employing aring interconnect and implementing the distributed home agentarchitecture of FIG. 2 b;

FIG. 4 shows the layers of the KTI protocol stack;

FIG. 5 is a schematic diagram illustrating the structure of a KTI link;

FIG. 6 shows a 192-bit flit employed in one embodiment of a KTI link;

FIG. 7 depicts an exemplary HTID allocation table;

FIG. 8 depicts details of an exemplary route-back table;

FIG. 9 shows a flowchart illustrating operations performed in connectionwith a raw memory READ transaction, according to one embodiment;

FIGS. 10 a-c show exemplary configurations of packets used to facilitatememory transactions, according to one embodiment; and

FIG. 11 shows an exemplary leaderless packet comprising three 192-bitKTI flits.

DETAILED DESCRIPTION

Embodiments of methods, systems, and apparatus for implementing rawmemory transactions and associated protocols are described herein. Inthe following description, numerous specific details, such asimplementations employing Keizer Technology interconnect (KTI)interconnects and protocols, are set forth to provide a thoroughunderstanding of embodiments of the invention. One skilled in therelevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, materials, etc. In other instances, well-knownstructures, materials, or operations are not shown or described indetail to avoid obscuring aspects of the invention.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

For clarity, individual components in the Figures herein may also bereferred to by their labels in the Figures, rather than by a particularreference number. For example, the labeling of the nodes in variousFigures provides information identifying the node and/or its function;such information cannot be conveyed alone with separate referencenumbers. Additionally, reference numbers referring to a particular typeof component (as opposed to a particular component) may be shown with areference number followed by “(typ)” meaning “typical.” It will beunderstood that the configuration of these components will be typical ofsimilar components that may exist but are not shown in the drawingFigures for simplicity and clarity.

FIG. 1 is a block schematic diagram illustrating selected components andinterfaces according to a system architecture 100. Most of thecomponents for the system architecture are implemented on an SoC 102,which depicts an abstracted block-level view of selected components andfunctional blocks common to many SoC architectures. These componentsinclude multiple processor cores 104, which provide the primaryprocessing operations of the SoC. While the exemplary configurationshown in SoC 102 depicts 8 processor cores, it will be recognized thatSoC 102 could include various number of processor cores, such as 1, 2,4, 6, 8, 10, 12, etc. Each of processor cores 104 are connected to arouter block 106 via an internal interconnect 108. The router block 106and internal interconnect 108 are generally representative of variouscircuitry that supports communication between components in SoC 102,including busses, serial point-to-point links, and interconnect fabrics,as applicable. Further details of such connections are not shown so asto not obscure the detail of system architecture 100.

A typical SoC will include various interfaces for communication withcomponents external to the SoC, such as disk drives and otherinput/output (I/O) devices, network interfaces, BIOS/firmware, andperipheral devices, as well as other SoC's that may be coupled to SoC102 via CPU socket-to-socket interconnects or other forms ofinterconnects used for communication between SoCs. Some of theinterfaces for facilitating communication to the external componentsinclude various PCIe interconnects, generally depicted as double-arrowedx16 PCIe interfaces 112 with a corresponding box labeled x16 (indicatinga link width of 16 lanes), and x8 PCIe interfaces 113 with acorresponding box labeled x8 (indicating a link width of 8 lanes).However, the link widths and numbers of the PCIe interfaces are merelyexemplary, as the actual links may be between 1 and 32 (x32) lanes wide.The PCIe interfaces are used for interfacing with various peripheral andsystem components, such as PCIe expansion slots, video cards, videochips, etc.

SoC 102 also includes various fabric interfaces (I/F) 114. In general,fabric interfaces may be connected to an interconnect fabric, such asinterface fabric 116 shown in FIG. 1, or to one or more point-to-pointinterconnect links. Although interface 116 is shown as a single blockfor convenience, it will be understood that a given fabric interface 116may be connected to an interconnect fabric or link that is specific toonly that fabric interface, or to an interconnect fabric that isaccessible via other one or more other fabric interfaces.

The remaining components shown in system architecture 100 pertain tomemory access, caching, and coherency. It is also common to havemultiple levels of caches, with caches closest to the processor corehaving the least latency and smallest size, and the caches further awaybeing larger but having more latency. For example, a typicalconfiguration might employ first and second level caches, commonlyreferred to as L1 and L2 caches. Another common configuration mayfurther employ a third level or L3 cache.

In the context of system architectures disclosed herein, the highestlevel cache is termed the Last Level Cache, or LLC. For example, the LLCfor a given core may typically comprise an L3-type cache if L1 and L2caches are also employed, or an L2-type cache if the only other cache isan L1 cache. Of course, this could be extended to further levels ofcache, with the LLC corresponding to the last (i.e., highest) level ofcache.

The caches shown in architecture 100 include a first level (L1) and asecond level (L2) cache (as depicted by L1/L2 blocks 118) that are“private” to each processor core 104. Each of processor cores 104 isalso connected to an L3 cache comprising a last level cache (LLC) 120.Last level cache 120 is depicted as a single logical block in FIG. 1;however, as explained in further detail below, components relating tolast level cache 120 may be distributed on the SoC rather thanimplemented as a single monolithic block or box.

SoC 102 also includes a caching agent 122 coupled to a coherent agent124 (also referred to as a “home” agent), which in turn is connected toa memory controller 126. The caching agent 122, home agent 124, andmemory controller 126 work in concert to manage access to system memory128, which is shown as comprising four memory blocks 0-3. Memory blocks0-3 represent a logical partitioning of memory resources that areaccessed via memory controller 126. The actual physical memory is storedon one or more memory modules 130 and accessed from memory controller126 via a memory interface 132. For example, in one embodiment memoryinterface 132 includes one or more DDR interfaces, such as DDR3interfaces.

Home agent 124 interacts with caching agent 122 to manage cache lineusage by the various memory consumers (e.g., processor cores 104). Inparticular, these entities support a coherent memory scheme under whichmemory can be shared in a coherent manner without data corruption. Tosupport this functionality, home agent 124 employs a cache filter 134,and the caching and home agent access and update cache line usage datastored in a directory 136, which is logically depicted in memorycontroller 126 (i.e., part of the logic employed by memory controllerrelates to usage of directory 136 data) but whose data will typically bestored in system memory 128.

FIG. 2 a shows further details of selected memory-related components ofFIG. 1, while FIG. 3 a shows an exemplary SoC architecture 300 in whichthese components may be implemented. As depicted in FIG. 2 a, cachingagent 122 is shown to be divided into a plurality of “slices.” Asdiscussed below, caching agent 122 may be implemented in a distributedmanner under which each caching agent slice is associated with acorresponding processor core and LLC “slice” and supports cachecoherency operations associated with memory accesses for that core. Atthe same time, the plurality of caching agent slices work cooperativelyin a manner under which the caching agent functionality is managed on asystem-wide level.

FIG. 2 a also shows further details of memory 128, including multiplecache lines 138. Each cache line is further depicted to include an errorcorrection code (ECC) portion. In one embodiment, each cache linecomprises 64 data bits plus 8 ECC bits for a total of 72 bits. Access tothe cache lines is managed by a combination of caching agents, homeagent 138, and memory controller 126 such that memory coherence ismaintained.

FIG. 3 a shows a system 300 under which the memory management schemes ofFIGS. 1 and 2 a may be implemented, according to one embodiment. System300 is illustrative of an advanced system architecture implemented in anSoC 302 including multiple processor cores 304, each coupled to arespective node 306 on a ring interconnect 308. For simplicity, thenodes for ring interconnect 308 is shown being connected with a singleline. As shown in detail 310, in one embodiment each of these ringinterconnects include four separate sets of “wires” or electronic pathsconnecting each node, thus forming four rings for ring interconnect 308.In actual practice, there are multiple physical electronic pathscorresponding to each wire that is illustrated. It will be understood bythose skilled in the art that the use of a single line to showconnections herein is for simplicity and clarity, as each particularconnection may employ one or more electronic paths.

Each node labeled CBo n (where n is a number) is a node corresponding toa processor core sharing the same number n (as identified by the core'sengine number n). In the exemplary architecture of FIG. 3 a, there areeight processor cores including processor engines 0-7. However, this ismerely exemplary, as the number of processor cores (and associatedcomponents described below) may generally comprise two or more processorcores. As with the processor cores of FIG. 1, each of processor cores304 include an in L1/L2 cache 312 and a processor engine 314. Alsoconnected to each CBo n node is an LLC cache “slice” 316 and a cachingagent 318.

in the illustrated configuration of FIG. 2, each processor core 204includes a processing engine 312 coupled to an L1 or L1/L2 (as shown)cache 314, which are “private” to that core. Also connected to each CBon node and collocated with a respective processor core is an LLC cache“slice” 316 of a distributed LLC 120 and a caching agent 318. Under thisdistributed LLC scheme, each of the other processor cores has access toall of the distributed LLC cache slices. Under one embodiment, thedistributed LLC is physically distributed among N cores using N blocksdivided by corresponding address ranges. Under this distribution scheme,all N cores communicate with all N LLC slices, using an address hash tofind the “home” slice for any given address. Suitable interconnectcircuitry is employed for facilitating communication between the coresand the slices; however, such circuitry is not show in FIG. 3 a forsimplicity and clarity.

There are also other types of nodes shown in SoC 302 including KTI nodes0/1 and 2, an IIO node, a PCIe node, and a home agent (HA) node 0. KTInode 0/1 is depicted as being coupled to a fabric interconnect 320.Moreover, in one embodiment, KTI node 0/1 provides KTI interfacecircuitry forming two separate KTI interfaces. The IIO node isoperatively coupled to an Input/Output interface 322. Further shown area number of nodes marked with an “X”; these nodes are used for timingpurposes. It is noted that the KTI, IIO, PCIe and X nodes are merelyexemplary of one implementation architecture, whereas otherarchitectures may have more or less of each type of node or none at all.Moreover, other types of nodes (not shown) may also be implemented.

In one embodiment, data is passed between nodes in a cyclical manner.For example, for each real or logical clock cycle (which may span one ormore real clock cycles), data is advanced from one node to an adjacentnode in the ring. In one embodiment, various signals and data may travelin both a clockwise and counterclockwise direction around the ring. Ingeneral, nodes 306 may comprise buffered or unbuffered nodes. In oneembodiment, at least some of nodes 306 are unbuffered.

Each of caching agents 318 is configured to perform messaging relatingto signal and data initiation and reception in connection with acoherent cache protocol implemented by the system, wherein the cachingagent handles cache-related operations corresponding to addresses mappedto its collocated LLC 316. In addition, in one embodiment home agent HA0employs respective a filter 134, and the various caching and home agentsaccess and update cache line usage data stored in a directory 136 thatis implemented in a portion of memory 128. It will be recognized bythose skilled in the art that other techniques may be used formaintaining information pertaining to cache line usage.

In the context of system 300, a cache coherency scheme may beimplemented by using independent message classes. Under one embodimentof a ring interconnect architecture, independent message classes may beimplemented by employing respective wires for each message class. Forexample, in the aforementioned embodiment, ring interconnect 308includes four ring paths or wires, labeled and referred to herein as AD,AK, IV, and BL. Accordingly, since the messages are sent over separatephysical interconnect paths, they are independent of one another from atransmission point of view.

FIGS. 2 h and 3 b show details of a system 300A comprising anaugmentation of system 300 of FIGS. 2 a and 3 a that employs distributedhome agents 324 on an SoC 302A. As shown in FIG. 3 b, each home agent324 is collocated with a respective caching agent 318 and LLC 316.Similarly, each home agent 324 is operationally coupled to a respectiveprocessor core 304. In one embodiment, caching agents and home agentsare implemented via a common circuit block or box, or are otherwiseconsidered as a single logical component, which is referred to as acaching home agent or CHA (not shown).

Distributed home agents 324 are configured for performing memorycoherency operations that are analogous to home agent 124 discussedabove. However, in this instance, each home agent 324 is mapped to aportion or “slice” of memory (rather than a larger block), and that homeagent is responsible for performing home agent operations associatedwith that portion of memory. Accordingly, the filtering function of thehome agent is also distributed such that each home agent 324 includes afilter 326. In one embodiment, the filter data is stored the collocatedLLC 316. In one embodiment, the portion of memory associated with a homeagent comprises a portion of memory accessed by a single memorycontroller. In another embodiment, a home agent may be associated withmultiple portions of memory accessed via respective memory controllers,e.g., a first portion accessed via a first memory controller, a secondportion accessed via a second memory controller, etc.

Distribution and collocation of home agents 324 enables additionalfunctionality to be implemented in architecture 300A, including accessto memory controllers that may be off-chip (e.g., separate from SoC302). Under one embodiment, this is facilitated by use of KTI linksbetween KTI nodes on ring interconnect 308 and applicable memorycontrollers and KTI agents. For example, this is depicted in FIG. 3 b asKTI links 326 and 328 between KTI node 2 and a KTI node 3 (which hasreplaced node HA0 in SoC 302A). Each of KTI links 326 and 328 links itsKTI node with a respective memory controller 126A and 126B. Each ofmemory controllers 126A and 126B is used for controlling a respectivememory 128A and 128B. In one embodiment, each of memory 128A and 128Bincludes a respective directory 136A and 136B. Under an alternativeapproach, directory information may be managed by a cache agent or ahome agent rather than a memory controller.

As each of the processor cores executes its respective code, variousmemory accesses will be performed. The majority of these memory accesseswill comprise memory READ operations. Moreover, for a given processorcore, a significant portion of the memory READ operations will entailreading data stored in a system memory store, as opposed to readingmemory that is stored in a cache level associated with another processorcore. For example, well-architected applications and operating systemswill employ a single thread for performing memory READ operationsassociated with a given document that is primarily targeted for viewingrather than editing, such as a PDF document or web page. This processingwill typically involve either downloading the document data into amemory buffer (typically allocated as a portion of system memory) orreading document data from a document stored on a disk drive into amemory buffer. In turn, corresponding data will be “read” by theapplication thread by performing memory READ operations that enable thedata to be accessed by the processor core associated with the thread.Similarly, memory READ operations are employed to load application codeinto processor core registers to be executed.

The net result of the foregoing is memory READ operations of contentstored in system memory resources consume a significant portion ofsystem I/O bandwidth. In addition, each memory READ operation hasassociated overhead that results in addition bandwidth consumption,particularly when considering the need for routing data to differentcomponents that may share access to the same memory, such as is commonwith multi-core, multi-level cache architectures.

In accordance with aspects of embodiments disclosed herein, techniquesfor performing enhanced READ memory access, referred to as raw memorytransactions, are facilitated by employing “leaderless” packets that arerouted to appropriate agents through the use of augmentations to the KTIprotocol and corresponding KTI agents and interfaces. In further detail,memory transfers between KTI nodes and memory controllers areimplemented via packet-based transactions without the use of the packetheaders, thus freeing up the portion of a packet format normallyreserved for routing information so that it may be used for requestedmemory payload data. To better understand how this may be implemented inembodiments employing KTI, an overview of KTI operations is nowpresented.

Overview of Keizer Technology Interconnect

Keizer Technology Interface (KTI) is a recently developed interface andassociated protocols that leverages some aspects of QPI and providesextended functionality. As with QPI, KTI transactions are facilitatedvia packetized messages transported over a multi-layer protocol. Asshown in FIG. 4, the layers include a Physical layer, a Link layer, aTransport layer, and a Protocol layer. At the Physical layer, data isexchanged in phits (Physical Units). At the link layer phits areaggregated into flits (flow control units). At the Protocol layer,messages are transferred between agents using a packet-based transport.

The Physical layer defines the physical structure of the interconnectand is responsible for dealing with details of operation of the signalson a particular link between two agents. This layer manages datatransfer on the signal wires, including electrical levels, timingaspects, and logical issues involved in sending and receiving each bitof information across the parallel lanes. As shown in FIG. 5, in oneembodiment the physical connectivity of each interconnect link is madeup of twenty-four differential signal pairs plus a differentialforwarded dock. Each differential signal pair comprises a “lane,” andthus the embodiment of FIG. 5 employs 24 lanes. The user of 24 lanes isnot limiting, as KTI implementation may employ other numbers of lanes,including 6, 8, 12, and 16 lanes. Each port supports a link pairconsisting of two uni-directional links to complete the connectionbetween two components. This supports traffic in both directionssimultaneously.

Components with KTI ports communicate using a pair of uni-directionalpoint-to-point links, defined as a link pair, as shown in FIG. 5. Eachport comprises a Transmit (Tx) link interface and a Receive (Rx) linkinterface. For the illustrated example, Component A has a Tx port thatis connected to Component B Rx port. One uni-directional link transmitsfrom Component A to Component B, and the other link transmits fromComponent B to Component A. The “transmit” link and “receive” link isdefined with respect to a specific KTI agent. The Component A transmitlink transmits data from Component A Tx port to Component B Rx port.This same Component A transmit link is the Port B receive link.

The second layer up the protocol stack is the Link layer, which isresponsible for reliable data transmission and flow control. The Linklayer also provides virtualization of the physical channel into multiplevirtual channels and message classes. After the Physical layerinitialization and training is completed, its logical sub-block worksunder the direction of the link layer, which is responsible for flowcontrol. From this link operational point onwards, the logical sub-blockcommunicates with the Link layer at a flit granularity and transfersflits across the link at a phit granularity. A flit is composed ofintegral number of phits, where a phit is defined as the number of bitstransmitted in one unit interval (UI).

In one embodiment shown in FIG. 6 employing 24 lanes, a phit comprises24 bits, and a corresponding flit comprises 8 UI's for a total of 192bits. In general, each KTI flit comprises 192 bits, with the number ofbits in a phit and the number of LA's being a function of the number oflanes implemented for the KTI link.

The Routing layer is responsible for ensuring that messages are sent totheir proper destinations, and provides the framework for directingpackets through the interconnect fabric. If a message handed up from theLink layer is destined for an agent in another device, the Routing layerforwards it to the proper link to send it on. All messages destined foragents on the local device are passed up to the protocol layer.

The Protocol layer serves multiple functions. It manages cache coherencefor the interface using a write-back protocol. It also has a set ofrules for managing non-coherent messaging. Messages are transferredbetween agents at the Protocol level using packets. The Protocol layermanages delivery of messages across multiple links, involving multipleagents in multiple devices. The system's cache coherency acrossdistributed caches and memory controllers is maintained by distributedagents that participate in coherent memory space transactions, subjectto rules defined by the Protocol layer. The KTI coherency protocolsupports home snoop coherency schemes.

Raw Memory Transactions

As discussed above, home agents are employed to support coherent memorymanagement, including maintaining cache line status and cache linelocation for the memory address range(s) allocated to each cache agent.Under architectures employing a single home agent per SoC, updating ofcache line status and location is managed by a single entity and thuscache line usage information pertaining to memory managed by the homeagent are routed to the single home agent. Based on the home agent cacheline status and location information (i.e., where cached copies of thecache line may be located), the corresponding cache line data isretrieved from an appropriate source (either memory or an applicablecache) and a copy of the cache line is returned to the originalrequester, also referred to herein as the requesting entity. Thisrequires routing the cache line data back to the requester, which isfacilitated using KTI routing techniques described above. However, inaccordance with the embodiments of FIGS. 2 h and 3 b, the home agentoperations are distributed in a manner that co-locates a home agent witha respective CBo (and corresponding processor core) and caching agent.As a result, rather than forwarding or otherwise maintaining cache lineusage information at a single home agent, now such information isprovided to or managed by a distributed set of home agents. At the sametime, since the distributed home agents are co-located with processingcores at the CBo nodes, it is possible to tie the applicable home agentsto corresponding memory transactions originating for CBo's and/orco-located processor cores.

One aspect of distributing and collocating home agents with cachingagents and LLC slices is that transactions between a home agent and amemory controller (corresponding to cache lines managed by the homeagent) do not need to include a routing address to the node to which theLLC is collocated. Rather, only the home agent needs to be identified,as explained in further detail below.

In one embodiment, routing information is implemented such thatrequested memory READ data is returned to the home agent of anapplicable requestor without having to explicitly provide the routinginformation in the memory READ packets returned by a memory controller.This is facilitated, in part, through the use of a “route-back” table(RBT). The RIB contains home agent and CBo mapping information, alongwith other information used to generate Home Tracker Identifiers(HTIDs), which are included in the memory transaction requests andencoded such that the appropriate home agent to which a memory READrequest result is to be returned can be readily identified.

FIG. 7 shows an exemplary HTID allocation pool for a CBo (in thisexample CBo0). In the illustrated embodiment, each entry comprises an8-bit vector, with each bit representing whether or not an HTID isavailable for use (1) or not available (0). The use of an 8-bit vectoris merely exemplary, as other length bit vectors may be employed,depending on the number of home agents and/or memory controllers, andthe mappings between home agents and memory controllers.

FIG. 8 show further details of an exemplary Hill) mapping. FIG. 8 showsa mapping for a home agent HAn to CBo's 0-3, with each CBo beingallocated an HTID pool of 8 HTIDs. For a given home agent, the HTIDswill be unique. The HTID is encoded to route incoming responses frommemory READ requests on a KTI link to the home agent that originated therequest.

In one embodiment there are two main modes—a static mode and a dynamicmode. Under the static mode, the RBI is pre-populated by the system BIOSduring boot up, with the field values being static. Under the dynamicmode, contents of the RBT may be changed during run-time, depending onusage.

One use case for the dynamic mode is direct-to-core transactions. Underdirect-to-core transactions, any core may send a request to any CBo.When using the direct-to-core mode, a KTI agent returns data directly tothe HA for a requesting core (via the applicable CBo). Accordingly, KTImodules are configured to support route back to HA's. In one embodiment,this is implemented through use of a Ring Stop ID field in the RBT entryto hold the Ring Stop ID of the HA.

Returning to FIG. 3 b, in one embodiment, a plurality of RBI instances330 (i.e., a copy of a route-back table applicable to a given CBo) aredepicted as being operatively coupled to a respective CBo. In addition,route-back tables 332 and 334 are operatively coupled to respectivenodes KTI 2 and KTI 3, in general, applicable HTID data is looked-up inan RBT by a requesting entity (e.g., CBo or processor core) and embeddedin an outbound memory transaction request. The HTID information is thenprocessed by the KTI agents at nodes KTI 2 and KTI 3 (as applicable) toeffect routing of memory request READ data returned from a memory sourcememory controller or cache) back to the requester via the ringinterconnect.

The flow for a memory READ request transaction, according to oneembodiment proceeds as follows. Referring to the flowchart of FIG. 9,the process starts in a block 900 with the caching agent for the memoryrequester (e.g., co-located processor core) to identify the home agentthat “owns” the cache line(s) associated with the memory transactionrequest and sends a packet on the ring interconnect to that home agent.As discussed above, the home agents are distributed, with each homeagent managing a corresponding portion of memory. Mapping datacorresponding to portion of memory managed by each home agent ismaintained by or otherwise accessible to each caching agent. Typically,the memory request will include an address or range of addressescorresponding to where the data is nominally located in a system memoryresource. This address or address range is then used as an input by thecaching agent to determine which home agent manages or “owns” thecorresponding cache line(s). Once this HA is identified, the cachingagent forwards the request in a packet that includes a destination ortarget address corresponding to that HA.

Upon receiving the packet, the home agent looks up information in itsdirectory to locate a valid copy of the cache line, as depicted in ablock 902. In general, a valid copy will be marked as E (Exclusive) or S(Shared) if using the MESI or MESIF cache coherency schemes. For thisexample it is presumed there are no copies of the cache lines within anycache level, and thus a copy of the cache line(s) needs to be retrievedfrom system memory.

For a request to access system memory, the home agent generates a KTIpacket corresponding to the memory request including an HTIDcorresponding to the request and sends the packet to the KTI agent atthe KTI node coupled to the memory controller associated with the homeagent. These operations are shown in a block 904, with a simplifiedillustration of the corresponding packet shown in FIG. 10 a. Recall thatin addition to partitioning (i.e., distributing) portions of memory tothe home agents, the portion of memory managed by each home agent isaccessed via a corresponding memory controller. That is, there is a 1:1mapping between each home agent and its associated memory controllerwhich is used to access the portion of memory managed by that homeagent. Since that memory controller is accessed via a single KTI node(and corresponding KTI agent), the home agent will be configured to sendmemory requests to that KTI node.

Next, in a block 906, the KTI agent at the KU node processes the packetand generates a raw memory READ request as a KTI packet including theHTID and sends it to the memory controller over the KTI link between theKU node and the memory controller. In response, in a block 908 the KTIagent at the memory controller returns cache line data corresponding tothe memory request as one or more “headerless” packets with an embeddedHTID back to the KTI agent at the KTI node. For example, FIG. 11 showsan example of a headerless packet comprising three 192-bit flits with anembedded HTID, while FIG. 10 b depicts an abstracted version of the sameheaderless packet.

In further detail, FIG. 1 depicts a headerless packet format 800,including KTI Flits 800-1, 800-2, and 800-3, via which a 64 byte cacheline can be transferred in accordance with one embodiment of a rawmemory transaction. One aspect of the headerless packet format is theencoding of the HTID associated with the transaction in lane L2. Asshown, the HTID comprises a 10-bit value, with bits [9:8] and [7:6]being encoded in KTI Flit 800-1, bits [5:4] being encoded in KTI Flit800-2, and bits [3:0] encoded in KTI Flit 800-3. Lane L2 furtherincludes the following encodings. Each of KTI Flits 800-1, 800-2, and800-3 include an JIB Or bit. Data Byte 31 is split across KTI Flits800-1 and 800-2 as shown. KTI Flit 800-3 further includes two directorybits and a poison bit. The 64 byte cache line data, absent Data Byte 31,is contained in lanes L3-L23, as shown. In addition, lanes L0 and L1 areused for 16 bit CRC's.

The packet of FIG. 11 is deemed headerless because it does not employ aheader as would be normally recognized in a KTI packet or other formatsused for packetized data. Rather, the HTID value is encoded across threeflits using lane L2. Since the smallest logical unit of data transferunder KTI is a flit (and thus the use of a header requires acorresponding flit), the headerless packet scheme provides a 33%improvement in data transfer rate since a full 64-byte cache line can betransferred using only three flits rather than the four flits that wouldbe required using a packet with a header.

Headerless data packets are detected by their format, whichsubstantially differs from other types of KTI packets employing headers.In order to perform proper actions, various bits associated with KTIpacket handling that are not present in the headerless data packets areimplied. Details of the various types and numbers of bits, including theimplied bits, for one embodiment of a headerless KTI packet are shown inTABLE 1 below.

TABLE 1 Headerless Data Flit 0-2 (and embedded header information bits)Data bits 512 CRC 48 IIB 3 Directory 2 Ack 0 Crd 0 Poison 1 RTID 10Total Bits 576 Additional implied bits in headerless data packet: Crd =VNA of 1 Ack = 1+ VNT = VNA MC/Opcde = MemData VN/NID unused in SMI3mode (filled as all zeros) Poison on First chunk is assumed clear.Poison cases must use full header.

Returning to the flowchart of FIG. 9, in a block 9010 the KTI agent atthe KTI node receives the requested data as one or more cache linestransferred as one or more respective headerless packets. The KTI agentthen decodes the HTID and uses the HTID as an input to its route-backtable to identify the corresponding home agent for the request. The KTIagent then generates a header including the home agent address in afourth flit that is pre-pended to the three other flits containing thecache line data and the corresponding packet is routed back to the homeagent via the ring interconnect. At this point the home agent carriesout various cache line coherency operations in accordance with the KTImemory coherency protocol.

First a core sends a request to its CA. In response, the CA checks itscache levels to see if the corresponding cache line(s) are alreadycached locally. If the cache line(s) are not in a local cache, therequest is passed to the HA that owns the corresponding cache line(s),which then generates a memory request packet and sends the request to anapplicable KTI agent (using the address of a corresponding KTI node).The KTI agent looks up the RBT and populates the entry's Ring Stop IDfield of the HA. The returning response performs the same look-up at theKTI agent and thus knows where to return the data. As a result, memoryREAD data returned from a memory controller coupled to a KTI node on thering does not have to include explicit routing information.

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of and examples for, the invention aredescribed herein for illustrative purposes, various equivalentmodifications are possible within the scope of the invention, as thoseskilled in the relevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification and the drawings. Rather, the scope ofthe invention is to be determined entirely by the following claims,which are to be construed in accordance with established doctrines ofclaim interpretation.

What is claimed is:
 1. A System on a Chip (SoC), comprising: a ringinterconnect having a plurality of nodes; a plurality of processorcores, each operatively coupled to a respective node; a plurality ofcaching agents, each operatively coupled to a respective processor core;a plurality of last level cache (LLC) slices, each LLC slice associatedwith a respective caching agent; a plurality of distributed home agents,each operatively coupled to a respective node; and a first interconnectlink interface, operatively coupled to a first interconnect interfacenode and configured to support an interface between the first interfacenode and a memory controller.
 2. The SoC of claim 1, wherein the firstinterconnect link interface is configured to interface to a memorycontroller that is external to the SoC.
 3. The SoC of claim 1, whereinthe first interconnect link interface employs a packet-based protocol tocommunicate with the memory controller.
 4. The SoC of claim 1, whereinthe packet-based protocol is based on the Keizer Technology Interconnect(KTI) protocol.
 5. The SoC of claim 1, wherein the SoC supports memorytransactions between the first interconnect link interface and a memorycontroller using a packet-based protocol implementing headerlesspackets.
 6. The SoC of claim 1, further comprising: a secondinterconnect link interface, operatively coupled to one of the firstinterconnect interface node or a second interface interface node andconfigured to support an interface between one of the first interconnectinterface node or a second interface interface node and a second memorycontroller.
 7. The SoC of claim 1, wherein a home agent and acorresponding caching agent are implemented as a single logicalcomponent.
 8. The SoC of claim 1, wherein the plurality of LLC slicesare managed as a single distributed last level cache.
 9. The SoC ofclaim 1, further comprising at least one route-back tables containinginformation for routing memory transaction request data back to a homeagent associated with a corresponding memory transaction.
 10. The SoC ofclaim 9, wherein a route-back table contains data including transactionidentifiers that are encoded to identify at least one of a home agent ornode to which the home agent is operatively coupled.
 11. A method,comprising: implementing a memory coherency protocol in a computersystem having a system on a chip (SoC) having a plurality of nodesconnected to form a ring interconnect, a portion of the plurality ofnodes having a processor core operatively coupled thereto; and employinga plurality of home agents to effect memory coherency operationsassociated with the memory coherency protocol, wherein each home agentis associated with a respective processor core.
 12. The method of claim11, further comprising implementing memory transactions using packetizedmessages between a link interface node comprising one of the nodes onthe ring interconnect and a memory controller external to the SoC andcommunicatively coupled to the link interface node via an interconnectlink.
 13. The method of claim 12, further comprising transferring memoryREAD request data corresponding to a memory READ request from the memorycontroller to the link interface node using at least one headerlesspackets.
 14. The method of claim 11, further comprising employing aroute-back table to effect routing of memory READ request data from amemory controller to a requesting entity that originates a correspondingmemory READ request.
 15. A computer system, comprising: A System on aChip (SoC), comprising: a ring interconnect having a plurality of nodes;a plurality of processor cores, each operatively coupled to a respectivenode; a plurality of caching agents, each operatively coupled to arespective processor core; a plurality of last level cache (LLC) slices,each LLC slice associated with a respective caching agent; a pluralityof home agents, each operatively coupled to a respective node andassociated with a respective processor core; a first interconnect linkinterface, operatively coupled to a first interconnect interface node onthe ring interconnect; and a first memory controller having a secondinterconnect link interface communicatively coupled to the firstinterconnect link interface via a first interconnect link.
 16. Thesystem of claim 16, wherein the first interconnect link interfaceemploys a packet-based protocol to communicate with the memorycontroller.
 17. The system of claim 16, wherein the packet-basedprotocol is based on the Keizer Technology Interconnect (KTI) protocol.18. The system of claim 16, wherein the SoC supports memory transactionsbetween the first interconnect link interface and the first memorycontroller using a packet-based protocol implementing headerlesspackets.
 19. The system of claim 16, wherein the SoC further comprises aplurality of route-back tables, each containing information for routingmemory transaction request data back to a home agent associated with acorresponding memory transaction.
 20. The system of claim 19, wherein aroute-hack table contains data including transaction identifiers thatare encoded to identify at least one of a home agent or node to whichthe home agent is operatively coupled.